System and method for multiple pass cooperative processing

ABSTRACT

A system for collaborative processing, comprising a controlling module with access to at least one relational database capable of performing a first set of functions on the data in the database and at least one external analytical engine, the external analytical engine being external to the relational database and being capable of a second set of functions on the data in the database. The controlling module is capable of iteratively processing a multi-step calculation including generating SQL statements to the relational database, passing preliminary results to an external analytical engine and saving data back into the relational database for further processing until the multi-step calculation is performed.

RELATED APPLICATIONS

This application is a continuation-in-part of U.S. application Ser. No.10/043,285 filed Jan. 14, 2002, entitled “System and method for multiplepass cooperative processing,” now abandoned, which is a continuation ofU.S. application Ser. No. 09/884,443, filed Jun. 20, 2001 entitled“System and method for multiple pass cooperative processing,” nowabandoned.

FIELD OF THE INVENTION

The invention relates to the field of data processing, and moreparticularly to the management of analytic processing against databasesto distribute processing tasks to necessary processing resources.

BACKGROUND OF THE INVENTION

The increase in enterprise software, data warehousing and otherstrategic data mining resources has increased the demands placed uponthe information technology infrastructure of many companies, academicand government agencies, and other organizations. For instance, a retailcorporation may capture daily sales data from all retail outlets in oneor more regions, countries or on a world wide basis. The resulting verylarge data base (VLDB) assets may contain valuable indicators ofeconomic, demographic and other trends.

However, databases and the analytic engines which interact with thosedatabases may have different processing capabilities. For instance, adatabase itself, which may be contained within a set of hard disk,optical or other storage media connected to associated servers ormainframes, may contain a set of native processing functions which thedatabase may perform. Commercially available database packages, such asSybase™, Informix™, DB2™ or others may each contain a different set ofbase functions. Those functions might include, for instance, thestandard deviation, mean, average, or other metric that may becalculated on the data or a subset of the data in the database.Conversely, the analytic engines which may communicate with and operateon databases or reports run on databases may contain a different, andtypically larger or more sophisticated, set of processing functions androutines.

Thus, a conventional statistical packages suck as the SPSS Inc. SPSS™ orWolfram Research Mathematica™ platforms may contain hundreds or more ofmodules, routines, functions and other processing resources to performadvanced computations such as regression analyses, Bayesian analyses,neural net processing, linear optimizations, numerical solutions todifferential equations or other techniques. However, when coupled to andoperating on data from separate databases, particularly but not limitedto large databases, the communication and sharing of the necessary ormost efficient computations may not always be optimized between theengine and database.

For instance, most available databases may perform averages on sets ofdata. When running averages on data, it is typically most efficient tocompute the average within the database, since this eliminates the needto transmit a quantity of data outside the database, compute thefunction and return the result. Moreover, in many instances the greatestamount of processing power may be available in the database and itsassociated server, mainframe or other resources, rather than in a remoteclient or other machine.

On the other hand, the analytic engine and the associated advancedfunctions provided by that engine may only be installed and available ona separate machine. The analytic engine may be capable of processing asuperset of the functions of the database and in fact be able to computeall necessary calculations for a given report, but only at the cost oflonger computation time and the need to pass data and results back andforth between the engine and database. An efficient design for sharedcomputation is desirable. Other problems exist.

SUMMARY OF THE INVENTION

The invention overcoming these and other problems in the art relates toa system and method for multipass cooperative processing whichdistributes and manages computation tasks between database resources,analytic engines and other resources in a data network. While othersystems have been capable of processing part of a SQL request in thedatabase and the other part in an analytical engine/process in a singledirection manner, various embodiments of the present invention providefor iterative, multi-directional processing of an entire report beingprocessed against the relational database system.

The present invention provides a process for handling multiple steps ina calculation iteratively between a controlling module, a database andan analytical engine external to the database. In this processingenvironment, some of the calculations or functions to be performed onthe data may be performed by the database itself and other calculationsor functions may be performed by the external analytical engine. Thecontrolling module resides outside of the relational database receives areport request or other non-SQL request. The controlling module monitorseach step in the processing of the report, acting as director over theactivities to maximize efficiency and handle complicated multi-sequencecalculations so that they do not result in an error.

The controlling module generates the SQL statement needed to be executedagainst the relational database. Upon generation of the SQL, thecontrolling module directs a first initial query to the database toresolve one step in the multi-step calculation (e.g., fetching,filtering, calculation or aggregate operations). The controlling modulethen generates a fetch operation to retrieve the data produced by theinitial query outside of the database (and the database's control). Thecontrolling module then passes at least some of the data produced by theinitial query to the external analytical engine to perform one or moreprocessing steps on the data. The controlling module then receives theprocessed results from the external analytical engine and transfers datafrom that result back into the originating database (e.g., in a databasetable) or some other database instance. Once in the originating databaseor the other database instance, the controlling module may direct thatfurther processing occur using the originating database that nowincludes the data processed by the database and external analyticalengine. To do so, the controlling module may generate another SQLstatement. That further processing may be done by the database and/ordata fetched and provided to the external analytical engine. These stepsmay continue in any order or sequence and as many times as desired untilall of the processes are completed, with the controlling enginegenerating SQL to perform various calculations or operations. Thus, thepresent invention allows for multiple levels of nested calculationsincluding calculations that may be performed by the database and thosethat may be performed by the external analytical engine.

The controlling module then provides the ability to pass the result backthe requesting system. Also, the controlling module may directprocessing to different databases so that various processes aretransmitted to other databases for storage or processing. Thus, in onesequence, data could be retrieved from database A, processed by externalanalytical engine 1, transmitted to database B, processed with data fromdatabase B, transmitted back into database A, processed again byexternal analytical engine 2, and then passed back to the requester.

In one embodiment of the invention, calculations native to a givendatabase platform nay be trapped and executed in the database, whileother types of functions are transmitted to external computationalresources for combination into a final result, such as a report executedon the database. In another regard, the invention may permit dataincluding intermediate results to be passed between the computingresources on a cooperative or collaborative basis, so that allcomputations may be located to their necessary or most efficientprocessing site. The exchange of data may be done in multiple passes.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the accompanyingdrawings, in which like elements are referenced with like numbers.

FIG. 1 is a block diagram illustrating an architecture for a systemaccording to an embodiment of the invention.

FIG. 2 is a flowchart illustrating steps performed by a processutilizing a query engine according to an embodiment of the invention.

FIG. 3 is a block diagram illustrating an architecture for a systemaccording to an embodiment of the invention.

FIG. 4 is a flowchart illustrating steps performed by a process fordistributed computation according to an embodiment of the invention.

FIG. 5 is a flowchart illustrating steps performed during distributedprocessing according to an embodiment of the invention, in anotherregard.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

FIG. 1 is a block diagram illustrating a system 100 by which a varietyof data resources may be accessed for business analytic, reportgeneration and other intelligence purposes according to an embodiment ofthe invention. According to a preferred embodiment, the system 100 maycomprise an Online Analytical Processing (OLAP) decision support system(DSS). In particular, FIG. 1 may comprise a portion of the MicroStrategy7 or 7.1 platform which provides a preferred system in which the presentinvention may be implemented.

In general, through using the system 100 of the invention, analysts,managers and other users may query or interrogate a plurality ofdatabases or database arrays to extract demographic, sales, and/orfinancial data and information and other patterns from records stored insuch databases or database arrays to identify strategic trends. Thosestrategic trends may not be discernable without processing the queriesand treating the results of the data extraction according to thetechniques performed by the systems and methods of the invention. Thisis in part because the size and complexity of some data portfoliosstored in such databases or database arrays may mask those trends.

In addition, system 100 may enable the creation of reports or servicesthat are processed according to a schedule. Users may then subscribe tothe service, provide personalization criteria and have the informationautomatically delivered to the user, as described in U.S. Pat. No.6,154,766 to Yost et al., which is commonly assigned and herebyincorporated by reference.

As illustrated in FIG. 1, a business, a government or another user mayaccess the resources of the system 100 using a user engine 102. The userengine 102 may include a query input module 116 to accept a plurality ofsearches, queries or other requests, via a query box on a graphical userinterface (GUI) or another similar interface. The user engine 102 maycommunicate with an analytical engine 104. The analytical engine 104 mayinclude a set of extensible modules to run a plurality of statisticalanalyses, to apply filtering criteria, to perform a neural net techniqueor another technique to condition and treat data extracted from dataresources hosted in the system 100, according to a query received fromthe user engine 102.

The analytical engine 104 may communicate with a query engine 106, whichin turn interfaces to one or more data storage devices 108 a, 108 b . .. 108 n (where n is an arbitrary number). The data storage devices 108a, 108 b . . . 108 n may include or interface to a relational databaseor another structured database stored on a hard disk, an optical disk, asolid state device or another similar storage media. When implemented asdatabases, the data storage devices 108 a, 108 b . . . 108 n may includeor interface to, for example, an Oracle™ relational database such assold commercially by Oracle Corporation, an Informix™ database, aDatabase 2 (DB2) database, a Sybase™ database, or another data storagedevice or query format, platform or resource such as an OLAP format, aStandard Query Language (SQL) format, a storage area network (SAN), or aMicrosoft Access™ database. It should be understood that while datastorage devices 108 a, 108 b . . . 108 n are illustrated as a pluralityof data storage devices, in some embodiments the data storage devicesmay be contained within a single database or another single resource.

Any of the user engine 102, the analytical engine 104 and the queryengine 106 or other resources of the system 100 may include or interfaceto or be supported by computing resources, such as one or moreassociated servers. When a server is employed for support, the servermay include, for instance, a workstation running a Microsoft Windows™NT™ operating system, a Windows™ 2000 operating system, a Unix operatingsystem, a Linux operating system, a, Xenix operating system, an IBM AIX™operating system, a Hewlett-Packard UX™ operating system, a NovellNetware™ operating system, a Sun Microsystems Solaris™ operating system,an OS/2™ operating system, a BeOS™ operating system, a MacIntoshoperating system, an Apache platform, an OpenStep™ operating system, oranother similar operating system or platform. According to oneembodiment of the present invention, analytical engine 104 and queryengine 106 may comprise elements of an intelligence server 103.

The data storage devices 108 a, 108 b . . . 108 n may be supported by aserver or another resource and may, in some embodiments, includeredundancy, such as a redundant array of independent disks (RAID), fordata protection. The storage capacity of any one or more of the datastorage devices 108 a, 108 b . . . 108 n may be of various sizes, fromrelatively small data sets to very large database (VLDB)-scale datasets, such as warehouses holding terabytes of data or more. The fieldsand types of data stored within the data storage devices 108 a, 108 b .. . 108 n may also be diverse, and may include, for instance, financial,personal, news, marketing, technical, addressing, governmental,military, medical or other categories of data or information.

The query engine 106 may mediate one or more queries or informationrequests from those received from the user at the user engine 102 toparse, filter, format and otherwise process such queries to be submittedagainst the data contained in the data storage devices 108 a, 108 b . .. 108 n. Thus, a user at the user engine 102 may submit a queryrequesting information in SQL format, or have the query translated toSQL format. The submitted query is then transmitted via the analyticalengine 104 to the query engine 106. The query engine 106 may determine,for instance, whether the transmitted query may be processed by one ormore resources of the data storage devices 108 a, 108 b . . . 108 n inits original format. If so, the query engine 106 may directly transmitthe query to one or more of the resources of the data storage devices108 a, 108 b . . . 108 n for processing.

If the transmitted query cannot be processed in its original format, thequery engine 106 may perform a translation of the query from an originalsyntax to a syntax compatible with one or more of the data storagedevices 108 a, 108 b . . . 108 n by invoking a syntax module 118 toconform the syntax of the query to standard SQL, DB2, Informix™, Sybase™formats or to other data structures, syntax or logic. The query engine106 may likewise parse the transmitted query to determine whether itincludes any invalid formatting or to trap other errors included in thetransmitted query, such as a request for sales data for a future year orother similar types of errors. Upon detecting an invalid or anunsupported query, the query engine 106 may pass an error message backto the user engine 102 to await further user input.

When a valid query such as a search request is received and conformed toa proper format, the query engine 106 may pass the query to one or moreof the data storage devices 108 a, 108 n . . . 108 n for processing. Insome embodiments, the query may be processed for one or more hitsagainst one or more databases in the data storage devices 108 a, 108 b .. . 108 n. For example, a manager of a restaurant chain, a retail vendoror another similar user may submit a query to view gross sales made bythe restaurant chain or retail vendor in the State of New York for theyear 1999. The data storage devices 108 a, 108 b . . . 108 n may besearched for one or more fields corresponding to the query to generate aset of results 114.

Although illustrated in connection with each data storage device 108 inFIG. 1, the results 114 may be generated from querying any one or moreof the databases of the data storage devices 108 a, 108 b . . . 108 n,depending on which of the data resources produce hits from processingthe search query. In some embodiments of the system 100 of theinvention, the results 114 may be maintained on one or more of the datastorage devices 108 a, 108 b . . . 108 n to permit one or morerefinements, iterated queries, joins or other operations to be performedon the data included in the results 114 before passing the informationincluded in the results 114 back to the analytical engine 104 and otherelements of the system 100.

When any such refinements or other operations are concluded, the results114 may be transmitted to the analytical engine 104 via the query engine106. The analytical engine 104 may then perform statistical, logical orother operations on the results 114 for presentation to the user. Forinstance, the user may submit a query asking which of its retail storesin the State of New York reached $1M in sales at the earliest time inthe year 1999. Or, the user may submit a query asking for an average, amean and a standard deviation of an account balance on a portfolio ofcredit or other accounts.

The analytical engine 104 may process such queries to generate aquantitative report 110, which may include a table or other outputindicating the results 114 extracted from the data storage devices 108a, 108 b . . . 108 n. The report 110 may be presented to the user viathe user engine 102, and, in some embodiments, may be temporarily orpermanently stored on the user engine 102, a client machine orelsewhere, or printed or otherwise output. In some embodiments of thesystem 100 of the invention, the report 110 or other output may betransmitted to a transmission facility 112, for transmission to a set ofpersonnel via an email, an instant message, a text-to-voice message, avideo or via another channel or medium. The transmission facility 112may include or interface to, for example, a personalized broadcastplatform or service such as the Narrowcaster™ platform or Telecaster™service sold by MicroStrategy Incorporated or another similarcommunications channel or medium. Similarly, in some embodiments of theinvention, more than one user engine 102 or other client resource maypermit multiple users to view the report 110, such as, for instance, viaa corporate intranet or over the Internet using a Web browser. Variousauthorization and access protocols may be employed for security purposesto vary the access permitted users to such report 110 in suchembodiments.

Additionally, as described in the '766 Patent, an administrative leveluser may create a report as part of a service. Subscribers/users maythen receive access to reports through various types of data deliverydevices including telephones, pagers, PDAs, WAP protocol devices, email,facsimile, and many others. In addition, subscribers may specify triggerconditions so that the subscriber receives a report only when thatcondition has been satisfied, as described in detail in the '766 Patent.The platform of FIG. 1 may have many other uses, as described in detailwith respect to the MicroStrategy 7 and 7.1 platform, the details ofwhich will be appreciated by one of ordinary skill in the reporting anddecision support system art.

The steps performed in a method 200 for processing data according to theinvention are illustrated in the flowchart of FIG. 2. In step 202, themethod 200 begins. In step 204, the user may supply input, such as aquery or a request for information, via the user engine 102. In step206, the user input query may be preliminarily processed, for instance,to determine whether it includes valid fields and for other formattingand error-flagging issues. In step 208, any error conditions may betrapped and an error message presented to the user, for correction ofthe error conditions. In step 210, if a query is in a valid format, thequery may then be transmitted to the analytical engine 104.

In step 212, the analytical engine 104 may further process the inputquery as appropriate to ensure the intended results 114 may be generatedto apply the desired analytics. In step 214, the query engine 106 mayfurther filter, format and otherwise process the input query to ensurethat the query is in a syntax compatible with the syntax of the datastorage devices 108 a, 108 b 108 n. In step 216, one or more appropriatedatabases or other resources within the data storage devices 108 a, 108b . . . 108 n may be identified to be accessed for the given query.

In step 218, the query may be transmitted to the data storage devices108 a, 108 b . . . 108 n and the query may be processed for hits orother results 114 against the content of the data storage devices 108 a,108 b . . . 108 n. In step 220, the results 114 of the query may berefined, and intermediate or other corresponding results 114 may bestored in the data storage devices 108 a, 108 b . . . 108 n. In step222, the final results 114 of the processing of the query against thedata storage devices 108 a, 108 b . . . 108 n may be transmitted to theanalytical engine 104 via the query engine 106. In step 224, a pluralityof analytical measures, filters, thresholds, statistical or othertreatments may be run on the results 114. In step 226, a report 110 maybe generated. The report 110, or other output of the analytic or otherprocessing steps, may be presented to the user via the user engine 102.In step 228, the method 200 ends.

In an embodiment of the invention illustrated in FIG. 3, the user maywish to generate a report 110 containing different types of metrics,illustrated as a first metric 120 and a second metric 122. The firstmetric 120 might illustratively be, for instance, an average or mean ofa data set, such as sales or other data. The second metric 122 mightillustratively be, for instance, a standard deviation or an analyticaltreatment, such as a regression or other analysis. In this illustrativeembodiment, the first metric 120 may be computable by the data storagedevices 108 a, 108 b . . . 108 n and their associated hardware or by theanalytic engine 104, whereas the second metric 122 may be computable bythe analytic engine, only.

In this embodiment, a management module 124 may be invoked to manage thedistribution of the computation of the report 110 including first metric120 and second metric 122. For instance, the management module maymaintain a table of computable functions, processes, routines and otherexecutable treatments that the analytical engine 104, data storagedevices 108 a, 108 b . . . 108 n, query engine 106 and other resourcesin the network of the invention may perform. The management module 124may then associate available resource with the necessary computationsfor the given report 110, including in this instance the first metric120 and the second metric 122. The management module 124 may beconfigured to detect and place the computation of functions in the mostefficient processing resource available at the time. For instance, themanagement module 124 may be configured to always or by default tocompute functions that the data storage devices 108 a, 108 b . . . 108 nare capable of computing within those devices.

In this illustrative embodiment, the management module 124 may detectthe first metric 120 as being computable within the data devices 108 a,108 b . . . 108 n and direct the computation of that metric, such as anaverage or mean, therein. The management module 124 may likewise detectthe second metric 122 as being computable by the analytic engine 104,and direct the computation of that metric, such as standard deviation orother metric, in that engine.

The management module 124 may also detect dependencies in thecomputation of the first metric 120, second metric 122 or other metricsnecessary to the computation of the report 110. For instance, it may bethe case that the first metric 120 is a necessary input to thecomputation of the second metric 122. In that instance, the managementmodule 124 may defer the computation of the second metric 122 until thedata storage devices 108 a, 108 b . . . 108 n have completed thecomputation of the first metric 120. The first metric 120 may then betransmitted as intermediate results to the analytic engine 104, wherethat metric may be used to compute the second metric 122. To achieve thegreatest efficiencies of computation and communication, any intermediateresults of any computation may be temporarily stored or cached on thedata storage devices 108 a, 108 b . . . 108 n or other resources so thatfurther computations need not re-compute or retrieve those intermediatedata unnecessarily.

Likewise, when computations may be most efficiently performed by thedata storage devices 108 a, 108 b . . . 108 n and inputs from theanalytic engine 104 may be needed for those computations, the analyticengine 104 may transmit (or “push”) results to the data storage, 108 a,108 b . . . 108 n for combination and computation therein. Thus,according to the invention the analytic engine 104, the data storagedevices 108 a, 108 b . . . 108 n and other engines or resources of thenetwork may act in concert to distribute processing to the necessary ormost optimal node of the network, in a collaborative or cooperativefashion, rather then according to a one-directional processing flowwhere computations and results are merely retrieved (or “pulled”) fromthe data storage devices 108 a, 108 b . . . 108 n for downstreamprocessing elsewhere. Iterative, stepwise or otherwise collaborativecomputations may thus be carried out, according to the invention.

Overall processing according to an embodiment of the invention fordistributed function selection and processing is illustrated in FIG. 4.In step 402, processing begins. In step 404, a user query may bereceived to generate a desired report 110. In step 406, the managementmodule 124 may be initiated or invoked. In step 408, the managementmodule 124 or other resources may parse the query for necessarycomputations or functions to deliver the report 110.

In step 410, the management module 124 may determine which computationsor functions may be computable in the data storage 108 a, 108 b . . .108 n or other resources. In step 412, the management module maydetermine which computations or functions may be computable in theanalytic engine 104 or other resources. In step 414, the managementmodule 124 may identify any dependencies in the order of computationneeded to generate report 110.

In step 416, the management module 124 may transmit instructions, suchas SQL or other commands, to the analytic engine 104, the data storagedevices 108 a, 108 b . . . 108 n to execute functions, computations orother processing of data from the data storage devices 108 a, 108 b . .. 108 n and intermediate results in those distributed resources.Processing may be concurrent or sequential, as appropriate. In step 418,any intermediate results may be iterated or stored locally ortemporarily for more efficient retrieval, such as in storage devices 108a, 108 b . . . 108 n or elsewhere. In step 420, the results of thecomputations from the various resources may be combined. In step 422, areport 110 may be generated containing the desired types of metrics,such as first metric 120 and second metric 122. In step 424, processingends.

Aspects of the iterated collaboration noted in step 418 of FIG. 4described above are illustrated in the flowchart of FIG. 5. As shown inthat figure, in step 502, instructions may be transmitted to the queryengine 106, for instance as part of the generation of a report 110. Instep 504, the query engine 106 may execute one or more calculation onthe data storage devices, 108 a, 108 b . . . 108 n. In step 506, theanalytical engine 104 may extract the results of the one or morecalculation from the data storage devices 108 a, 108 b . . . 108 n andexecute one or more calculations from its function set on those results,after which a bulk insert of the results from the analytical engine 104into the data storage devices 108 a, 108 b . . . 108 n may be performed.In step 508, a report 110 may be assembled or generated from thecollaborative processing, and presented to the user or otherwise output.

The foregoing description of the invention is illustrative, andvariations in configuration and implementation will occur to personsskilled in the art. For instance, resources illustrated as singular maybe distributed amongst multiple resources, whereas resources illustratedas distributed may be combined, in embodiments. The scope of theinvention is accordingly to be limited only by the following claims.

What is claimed is:
 1. A system for collaborative processing,comprising: a controlling module with access to at least one relationaldatabase capable of performing a first set of functions on the data inthe database and at least one external analytical engine, the externalanalytical engine being external to the relational database and beingcapable of a second set of functions on the data in the database;wherein the controlling module performs the following steps: receiving arequest to generate a report based on data in the relational database,the request including at least one multi-step calculation to beperformed on data in the relational database; generating a first SQLstatement to resolve a first step of the multi-step calculation on firstdata in the relational database; receiving results from the first SQLstatement; passing data generated by the first SQL statement to theexternal analytical engine, wherein the external analytical engine isdirected to perform at least one operation on the data; receivingexternally-operated data from the external analytical engine after theat least one operation; inserting the externally-operated data into arelational database; generating a second SQL statement to resolveanother step in the multi-step calculation, the second SQL statementoperating on at least part of the first data and at least part of theexternally-operated data; and generating a report in response to therequest after the second SQL statement has been resolved.
 2. The systemof claim 1 wherein the externally-operated data is inserted into the atleast one relational database in which the first data is stored.
 3. Thesystem of claim 1 wherein the externally-operated data is inserted intoa second relational database different from the relational database inwhich the first data is stored.
 4. The system of claim 1 wherein thecontrolling module performs the following additional steps: receivingresults from the second SQL statement; passing data generated by thesecond SQL statement to the external analytical engine, wherein theexternal analytical engine is directed to perform at least one operationon the data; receiving second externally-operated data from the externalanalytical engine after the at least one operation; inserting theexternally-operated data into a relational database; and generating athird SQL statement to resolve another step in the multi-stepcalculation, the third SQL statement operating on at least part of thefirst data and at least part of the second externally-operated data. 5.The system of claim 4 wherein the controlling module performs thereceiving, passing, receiving, inserting and generating steps at leastone additional time, thus generating a fourth SQL statement prior togenerating the report result.
 6. The system of claim 1 wherein the firstSQL statement comprises a fetch operation.
 7. The system of claim 1wherein the fist SQL statement comprises a filtering operation.
 8. Thesystem of claim 1 wherein the first SQL statement comprises acalculation operation.
 9. The system of claim 1 wherein the first SQL,statement comprises an aggregation operation.
 10. The system of claim 1wherein the operation performed by the external analytical engine is anoperation that the at least one relational database is incapable ofperforming.
 11. A method for collaborative processing in system withaccess to a relational database capable of performing a first set offunctions on the data in the database and at least one externalanalytical engine, the external analytical engine being external to therelational database and being capable of a second set of functions onthe data in the database, the method comprising the steps of: receivingat a controlling module external to a relational database a request togenerate a report based on data in the relational database, the requestincluding at least one multi-step calculation to be performed on data inthe relational database; generating a first SQL statement to resolve afirst step of the multi-step calculation on first data in the relationaldatabase; receiving results from the first SQL statement; passing datagenerated by the first SQL statement to the external analytical engine,wherein the external analytical engine is directed to perform at leastone operation on the data; receiving externally-operated data from theexternal analytical engine after the at least one operation; insertingthe externally-operated data into a relational database; generating asecond SQL statement to resolve another step in the multi-stepcalculation, the second SQL statement operating on at least part of thefirst data and at least part of the externally-operated data; andgenerating a report in response to the request after the second SQLstatement has been resolved.
 12. The method of claim 11 wherein theeternally-operated data is inserted into the at least one relationaldatabase in which the first data is stored.
 13. The method of claim 11wherein the eternally-operated data is inserted into a second relationaldatabase different from the relational database in which the first datais stored.
 14. The method of claim 11 further comprising the steps of:receiving results from the second SQL statement; passing data generatedby the second SQL statement to the external analytical engine, whereinthe external analytical engine is directed to perform at least oneoperation on the data; receiving second externally-operated data fromthe external analytical engine after the at least one operation;inserting the externally-operated data into a relational database; andgenerating a third SQL statement to resolve another step in themulti-step calculation, the third SQL statement operating on at leastpart of the first data and at least part of the secondexternally-operated data.
 15. The method of claim 14 wherein thereceiving, passing, receiving, inserting and generating steps areperformed at least one additional time, thus generating a fourth SQLstatement prior to generating the report result.
 16. The method of claim11 wherein the first SQL statement comprises a fetch operation.
 17. Themethod of claim 11 wherein the first SQL statement comprises a filteringoperation.
 18. The method of claim 11 wherein the first SQL statementcomprises a calculation operation.
 19. The method of claim 11 whereinthe first SQL statement comprises an aggregation operation.
 20. Themethod of claim 11 wherein the operation performed by the externalanalytical engine is an operation that the at least one relationaldatabase is incapable of performing.
 21. A machine readable medium, themachine readable medium being readable to execute a method forcollaborative processing, the method comprising: receiving at acontrolling module external to a relational database a request togenerate a report based on data in the relational database, the requestincluding at least one multi-step calculation to be performed on data inthe relational database; generating a first SQL statement to resolve afirst step of the multi-step calculation on first data in the relationaldatabase; receiving results from the first SQL statement; passing datagenerated by the first SQL statement to the external analytical engine,wherein the external analytical engine is directed to perform at leastone operation on the data; receiving externally-operated data from theexternal analytical engine after the at least one operation; insertingthe externally-operated data into a relational database; generated asecond SQL statement to resolve another step in the multi-stepcalculation, the second SQL statement operating on at least part of thefirst data and at least part of the externally-operated data; andgenerating a report in response to the request after the second SQLstatement has been resolved.